Display control device, display control method, program, and portable apparatus

ABSTRACT

The present invention provides a display control method and a display control device capable of detecting a position and an orientation of an apparatus by a simple structure and controlling a display of a displayed image based on the detected position and orientation of the apparatus with ease. Two cameras are mounted on the apparatus. A user holds the apparatus in the hand, captures, by a first camera, a first image including the face of the user, and captures, by a second camera, a second image including a scene opposite to the user. A display control unit extracts features of the face from the first image and extracts scene feature points from the second image. Then, the display control unit compares the extracted results with reference graphics and the like, and calculates a user-to-apparatus distance ro and angles θo, φo which indicate a direction to the apparatus, and controls the display of the displayed image based on the calculated distance ro and angles θo, φo.

CROSS REFERENCE TO RELATED APPLICATION

This is a continuation application of PCT application No.PCT/JP2005/009180 filed May 19, 2005, designating the United States ofAmerica.

BACKGROUND OF THE INVENTION

(1) Field of the Invention

The present invention relates to a display control method using imagemeasurement and image recognition, and particularly to a technologywhich detects a position and an orientation of an apparatus on which acamera is mounted, using an image captured by the camera, and controls adisplay based on the detected position and orientation of the apparatus.

(2) Description of the Related Art

In recent years, with the wide use of the Internet, the so-calledubiquitous network society is being formed, in which computers andhousehold electrical appliances are interconnected and remote control,media fusion and the like are realized. The society is characterized inhaving a high degree of flexibility which enables users to performcommunications at any time and any place, and the users can act withoutrecognizing differences in apparatuses and the physical distance betweenthe user and an apparatus. Although in the realization of the ubiquitousnetwork society, the network connection is an essential condition forthe system, a high degree of flexibility in the user interface betweenthe user and the apparatus is also a prerequisite. Even when anapparatus is connected as either hardware or software, unless the userwho uses the system can freely communicate with the system, it is hardto say being ubiquitous. To this end, the technology which enhances easein using a system, in other words, the technology which enhancesusability of the system is required.

In order to realize an intuitive operation method for enhancing theusability of a system, it is necessary for the system to automaticallyunderstand the intention of the user. For the realization, for example,a technology which detects positions and orientations of interfacedevices is useful.

One of such interface devices is a pointing marker. For example, in FIG.1 of Japanese Laid-Open Patent Application No. 2001-356875 (referred toas Patent Reference 1), a user holds, in a hand as a pointing marker, anobject (blue LED and the like) with brightness or color which isdifferent from that of the user. A position of the pointing marker isdetected by an image sensor in a system, and in accordance with thedetected position, a displayed image can be operated by the system. Withthis, the user can perform communications with the system intuitivelywith bodily sensation, without understanding the operation method of thesystem.

In addition, using brightness of a subject and color difference betweenthe subject and an image of the subject, there exists a method ofextracting a marker from a captured image (for example, refer to FIG. 1of Japanese Laid-Open Patent Application No. 2000-230806 which isreferred to as Patent Reference 2 hereinafter). The method is that acurrent marker position is compared to a reference marker positioncaptured in advance so as to detect a distance between the subject andthe image sensor. With the method, the user has only to register thereference marker by executing predetermined initial processing inactivating the system. Then, the system detects changes in the positionof the marker which are sequentially captured, and automatically obtainsthe distance between the subject and the image sensor.

In addition, the system, referred to as the virtual reality system,which gives the user sensation as if the user directly operated a realobject enables the user to directly control, without recognizing theapparatus, objects in virtual space which has been built using computergraphics. For example, when a portable display which the user holds inthe hand is moved three-dimensionally, the system detects the motion byan acceleration sensor and the like, and changes the displayed image toanother image according to the motion (for example, refer to FIG. 11 ofJapanese Patent Laid-Open Application No. 11-18025 which is referred toas Patent Reference 3 hereinafter). Thus, it is possible to intuitivelyprovide the system with information regarding changes in viewpoints inthe virtual reality space by changes in the position and orientation ofthe display held in the hand.

However, the technology described in Patent Reference 1 need to preparea special marker. Thus, there exists a problem in that the generalversatility lacks and situations where the technology is used arelimited. In other words, as the brightness of a subject and colordifference between the subject and an image of the subject are used fordetecting the marker, it is necessary to provide a marker with thebrightest color among colors of any other subjects and a color which isnot included in any other subjects.

In addition, with the technology described in Patent Reference 2, asingle image sensor tracks plural markers and detects a distance betweena subject and the image sensor. However, even with the technology, it isnot possible to detect a three-dimensional position of the image sensor.In other words, as the distance from the center of a sphere to anypoints on the sphere is all equal, such position information is regardedas the same. Thus, there remains a problem in the usability.

On the other hand, as the acceleration sensor or the like is used in thetechnology of Patent Reference 3, there is no need to capture themarker. Thus, it outperforms other system configurations in terms of thegeneral versatility. However, for example, as the acceleration sensorcan not detect a speed of uniform motion, there exists a problem interms of sensitivity of the sensor. To overcome this point, sensorfusion is effective which multi-directionally detectsposition/orientation information by a magnetometric sensor, a ultrasonicsensor or the like so as to complement the information each other.However, an additional sensor leads to increase in the cost of thesystem and increase in the capacity and weight of an apparatus. Inaddition, in many cases, such sensors constantly operate, and increasein the number of sensors leads to increase in electric power consumptionof the apparatus. In particular, this poses a serious problem toportable apparatuses.

Thus, the present invention has been conceived in view of theaforementioned circumstances, and the object is to provide a displaycontrol method and a display control device which can detect a positionand an orientation of an apparatus by a simple structure and control adisplay of a displayed image based on the detected position andorientation of the apparatus.

SUMMARY OF THE INVENTION

In order to achieve the aforementioned objects, the display controlmethod according to the present invention is a display control methodfor controlling a display of a displayed image based on a position of anapparatus provided with a first camera and a second camera which are ina known geometric relationship, and the display control method includes:capturing, by the first camera, a first image including an object with aknown image feature; capturing, by the second camera, a second imageincluding a scene in the vicinity of the object; extracting the imagefeature of the object from the first image; calculating a distance fromthe object to the apparatus and a direction of the apparatus viewed fromthe object, by comparing the extracted image feature of the object tothe known image feature of the object; extracting an image feature ofthe second image from the second image; calculating a distance from theobject to the apparatus and a direction of the apparatus viewed from theobject, by comparing the extracted image feature of the second image toan image feature extracted from a past image which has been captured bythe second camera; and controlling the display of the displayed imagebased on the distance and direction calculated based on the first imageand the distance and direction calculated based on the second image.

According to the present invention, as a position and an orientation ofan apparatus can be detected by images captured by two cameras, increasein the capacity and weight of the apparatus required for detection canbe prevented, and a display of a displayed image can be easilycontrolled. In addition, the electric power consumption can be reduced.

Further Information About Technical Background to this Application

The disclosure of Japanese Patent Application No. 2004-167242 filed onJun. 4, 2004 including specification, drawings and claims isincorporated herein by reference in its entirety.

The disclosure of PCT application No. PCT/JP2005/009180 filed, May 19,2005, including specification, drawings and claims is incorporatedherein by reference in its entirety.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other objects, advantages and features of the invention willbecome apparent from the following description thereof taken inconjunction with the accompanying drawings that illustrate a specificembodiment of the invention. In the Drawings:

FIG. 1 is a diagram showing a state of executing a display controlmethod according to the embodiment of the present invention.

FIG. 2 is a schematic diagram showing the positions of the user and thecellular phone represented by relative coordinates in a state of FIG. 1.

FIG. 3 is a schematic diagram showing representation of orientations ofan apparatus.

FIG. 4 is a block diagram showing the configuration of the displaycontrol device according to the embodiment of the present invention.

FIG. 5 is a diagram describing an example of a method for detecting adistance to the apparatus.

FIG. 6 is a diagram describing an example of a method for detecting adirection of the apparatus.

FIG. 7 is a diagram describing an example of a method for detecting anorientation of the apparatus.

FIG. 8 is a diagram describing the control of an enlargement display ora reduction display.

FIG. 9 is a diagram describing the control of a modified display to thedisplayed direction.

FIGS. 10 and 11 are variations of the configuration shown in FIG. 4.

DESCRIPTION OF THE PREFERRED EMBODIMENT

The display control method according to the present invention is adisplay control method for controlling a display of a displayed imagebased on a position of an apparatus provided with a first camera and asecond camera which are in a known geometric relationship, and thedisplay control method includes: capturing, by the first camera, a firstimage including an object with a known image feature; capturing, by thesecond camera, a second image including a scene in the vicinity of theobject; extracting the image feature of the object from the first image;calculating a distance from the object to the apparatus and a directionof the apparatus viewed from the object, by comparing the extractedimage feature of the object to the known image feature of the object;extracting an image feature of the second image from the second image;calculating a distance from the object to the apparatus and a directionof the apparatus viewed from the object, by comparing the extractedimage feature of the second image to an image feature extracted from apast image which has been captured by the second camera; and controllingthe display of the displayed image based on the distance and directioncalculated based on the first image and the distance and directioncalculated based on the second image.

With this, the distance from the object to the apparatus is calculatedfrom the first image which has been captured by the first camera andincludes the known image feature of the object, and in addition, thedirection of the apparatus which is viewed from the object is calculatedfrom the second image captured by the second camera. In other words, thedistance and direction of the corresponding apparatus with respect tothe object, in other words, the three-dimensional position of theapparatus can be detected by the images captured by the two cameras.Then, based on this three-dimensional position of the apparatus, it ispossible to easily control the display of the displayed image.

Here, it is preferable that the object is a face of a user who uses theapparatus. With this, the three-dimensional position of the apparatuswith respect to the position of the face of the user is calculated whenthe user just holds the apparatus in the hand, captures his/her face bythe first camera, and captures the scene in the vicinity of theapparatus by the second camera.

In addition, when the direction of the apparatus is calculated, thedirection of the apparatus may be calculated by comparing, to an imagefeature extracted from a past image which has been captured by thesecond camera, an image feature of the second image extracted from thesecond image. In addition, when the direction of the apparatus iscalculated, the direction of the apparatus may be calculated bycomparing, to a known image feature of the object, an image feature ofthe object extracted from the first image.

In addition, the display control method further includes calculating anorientation of the apparatus with respect to the object using at leastone of the image feature of the object extracted from the first imageand the image feature of the second image extracted from the secondimage, wherein controlling the display of the displayed image includescontrolling the display of the displayed image based on the calculateddistance from the object to the apparatus, the calculated direction ofthe apparatus, and the calculated orientation of the apparatus. Withthis, it is possible to easily control the display of the displayedimage in accordance with not only the three-dimensional position of theapparatus but also the orientation of the apparatus.

Note that the present invention can not only realize as such a displaycontrol method, but also as a display control device having thecharacteristic steps included in the display control method as units,and as a program which causes a computer to execute such steps. Inaddition, it is obvious that such program can be distributed via arecording medium, such as a CD-ROM and via a transmission medium, suchas the Internet.

The embodiment of the present invention is described hereinafter withreference to the diagrams.

FIG. 1 is a diagram showing a state of executing a display controlmethod according to the embodiment of the present invention. In FIG. 1,a cellular phone 20 as an apparatus of the present invention mounts auser-side camera 21 which captures an image of the user side as a firstcamera and an opposite-side camera 22 which captures an image oppositeto the user as a second camera, which are in a known geometricrelationship. Furthermore, in the diagram, an interactive display systemis realized which changes a displayed image to another image inaccordance with a position and an orientation of the cellular phone 20.

A user 10 holds the cellular phone 20 in the hand, and views an image onthe screen (the illustration is omitted). Among the two cameras 21 and22 which are mounted on the cellular phone 20, the user-side camera 21captures the face of the user 10 as an object in which the imagefeatures are known. On the other hand, the opposite-side camera 22shares an optical axis with the user-side camera 21, and captures animage which is opposite to the user. Therefore, the user 10 is notwithin the photographic angle of the opposite-side camera 22, and theopposite-side camera 22 captures a scene opposite to the user 10. Inaddition, a position/orientation detection unit 30 is mounted inside thecellular phone 20 as a position detection device, and thisposition/orientation detection circuit 30 detects the position andorientation of the cellular phone 20 which are determined by the motionof the hand of the user 10. On a screen, an image in accordance with theposition and orientation of the cellular phone 20 is displayed, and theuser 10 can freely select a desired image by moving the cellular phone20.

As a cellular phone has originally limitations in size of a display unit(a display) because of its portability, there is an upper limit in thesize of the image to be displayed. Thus, in the case where an image or apicture which is widely circulated for television and a computer isdisplayed on a display of a cellular phone, there exists a problem ofviewability in that the details are hard to read. To improve theviewability, the image has only to be enlarged with a center focus on aspecified point. Here, when the image enlargement is executedintuitively by changing the position or orientation of the cellularphone instead of pushing a button of the cellular phone or rotating adial of the cellular phone, the convenience is improved. In addition,after confirming the details, in the case where the whole image againneed to be viewed, when the size or a clipped area of the displayedimage can be directly changed in accordance with the position andorientation of the cellular phone, the user has the advantage over theoperability. The image change function in the interactive display systemcan be applied to various cases, such as a change in a shadow oflighting which illuminates a subject or in an orientation of thedisplayed subject, regardless of enlargement and reduction of the image.This significantly contributes to progress in expressiveness, reality,viewability and the like of images.

As shown in FIG. 1, the position of the cellular phone 20 is representedby a rotating coordinate system in which a point between the both eyesof the user 10 is designated as the rotation center. A distance from therotation center to the cellular phone 20 is indicated by ro, a rotationangle horizontal to a position coordinate normal line NL is indicated byθo, and a rotation angle vertical to the position coordinate normal lineNL is indicated by φo.

FIG. 2 is a schematic diagram showing the positions of the user 10 andthe cellular phone 20 represented by relative coordinates. As shown inFIG. 2, the closest position of the cellular phone 20 to the user atwhich the user can at least read the screen is assumed to be distancero=0.0, and the farthest position when the user's arm is fully extendedis assumed to be distance ro=1.0. In addition, within the range wherethe user 10 swings, from side to side, the arm which holds the cellularphone 20, the far-right position is assumed to be θo=1.0, the far-leftposition is assumed to be θo=−1.0, and a direction to the positioncoordinate normal line NL is assumed to be θo=0.0. Likewise, within therange where the user 10 swings, up and down, the arm which holds thecellular phone 20, the highest position is assumed to be φo=1.0, thelowest position is assumed to be φo=−1.0, and a direction to theposition coordinate normal line NL is assumed to be φo=0.0.

Note that the definition of the aforementioned position coordinates isone of the examples, and the present invention does not limit thedefinition method of the position coordinates.

FIG. 3 is a schematic diagram showing representation of orientations ofthe cellular phone 20. As shown in FIG. 3, an orientation of thecellular phone 20 (showed with a frame format in FIG. 3) is indicated byan angle horizontal to a reference plane and an angle vertical to thereference plane. The reference plane can be arbitrarily designated, andin FIG. 3, a plane RS is the reference plane which is determined by thepositions of the both eyes and the nose of the user 10 which arecaptured at an initial state. The orientation of the cellular phone 20in FIG. 3 is indicated by an angle θc which is an angle to thehorizontal direction with respect to the reference plane RS and an angleφc which is an angle to the vertical direction with respect to thereference plane RS. Note that the rotation center is not limited to thepoint between the both eyes, and it has only to be designated in thevicinity of the rotation center of the user 10.

FIG. 4 is a block diagram showing the configuration of the displaycontrol device according to the embodiment. The display control device30 in FIG. 4 calculates a position (ro, θo, φo) and an orientation (θc,φc) of the cellular phone 20 based on a first image IM1 that is an imagecaptured by the user-side camera 21 and a second image IM2 that is animage captured by the opposite-side camera 22, and controls a display ofthe displayed image. As shown in FIG. 4, the display control device 30is configured of a face feature extraction unit 31, a size comparisonunit 32 a, a position comparison unit 32 b, a shape comparison unit 32c, memory 33, a distance calculation unit 34, a direction calculationunit 35, a scene feature extraction unit 36, a size comparison unit 37a, a position comparison unit 37 b, a shape comparison unit 37 c, memory38, an orientation calculation unit 39 and a display control unit 41.Note that the first calculation unit includes the size comparison unit32 a, memory 33 and distance calculation unit 34, and the secondcalculation unit includes the position comparison unit 37 b, shapecomparison unit 37 c, memory 38 and direction calculation unit 35.

The face feature extraction unit 31 as the first feature extraction unitextracts face feature points from the first image IM1 with a centerfocus on face parts, such as the eyes, nose, and mouth, and outputs eachof face feature signals S1. The face feature signals S1 are supplied tothe size comparison unit 32 a, position comparison unit 32 b and shapecomparison unit 32 c respectively. In the memory 33, the face featurepoints extracted in advance are accumulated as reference graphics. Thescene feature extraction unit 36 as the second feature extraction unitextracts scene feature points from the second image IM2, and outputseach of scene feature signals S2. The scene feature signals S2 aresupplied to the size comparison unit 37 a, position comparison unit 37 band shape comparison unit 37 c respectively. In the memory 38, thefeature points extracted from the captured past images and informationof marker graphics are stored.

The size comparison unit 32 a compares the size of an image feature(marker graphics) obtained from the face feature signal S1 to the sizeof the reference graphics accumulated in the memory 33. The distancecalculation unit 34 calculates the distance ro from the comparisonresult obtained by the size comparison unit 32 a.

FIG. 5 is a diagram describing an example of a method for obtaining thedistance ro. In the example of FIG. 5, a face feature point isdesignated between the pupils, and line segments which connectrespective center of the pupils are designated as the marker graphics.In the memory 33, a line segment L1 which connects the center of thepupils obtained from an image A1 captured in a position where thedistance ro=0.0, and a line segment L2 which connects the center of thepupils obtained from an image A2 captured in a position where thedistance ro=1.0 are stored in advance as the reference marker graphics.The length of the line segment L1 is expressed by (de, near, base), andthe length of the line segment L2 is expressed by (de, far, base). Here,in the case where the length of the line segment (pupil distance) whichconnects the center of the pupils obtained from the face feature pointsignal S1 is assumed to be de, the distance ro is obtained by thefollowing equation.ro=(de,near,base−de)/(de,near,base−de,far,base)

As the image IM1 captured by the user-side camera 21 constantly includesthe face of the user 10, it can be expected that such a method whichdesignates a line segment which connects center of the pupils as markergraphics performs stable detection, compared to the extraction of markergraphics from an arbitrarily captured scene.

Although the marker graphics in the two positions where ro=0.0, 1.0 areused as initial data, the method for providing the initial data andcalculating the distance ro is not limited to the aforementioned method.For example, it may be possible to detect the pupils when initializingthe system after the turning-on of the power to the cellular phone,calculate the scaling of the pupil distance with respect to the distancebetween the pupils, and utilize the calculated scaling as the scaling ofthe distance ro.

In addition, although a line segment which connects the center of thepupils is used as marker graphics and the length of the line segment isused as the size of the marker graphics here, various kinds, sizes, orthe number of the marker graphics and various size definitions of suchmarker graphics can be conceived instead.

On the other hand, the effective way to detect a direction to thecellular phone 20, in other words, the angles θo and φo is to operatethe user-side camera 21 and opposite-side camera 22 in a coordinatedmanner. In FIG. 6, an image captured by the user-side camera 21 andopposite-side camera 22 is shown in the case where the cellular phone 20is moved to the horizontal direction (angle θo). In FIG. 6, a pencil, anotebook and a mug are put on a table in front of the user 10.

In other words, as the user-side camera 21 constantly captures the user10 which is the rotation center, even in the case where the cellularphone 20 is moved to the horizontal direction, it is possible to obtainalmost the same images captured by the user-side camera 21 (images C1,C2, and C3). The images C1 to C3 do not become completely the sameimage, because the cellular phone is moved by the hand and such motiondiffers from a mechanical rotation by a robot arm. Therefore, therespective positions of the marker graphics are tentatively differenteach other. However, as the difference between the marker graphics isnot so large, the resolution becomes low when the angle θo is detectedfrom the images C1 to C3.

On the other hand, as the opposite-side camera 22 captures a sceneopposite to the user 10, the obtained image largely varies depending onthe motions of the cellular phone 20, as shown in images B1, B2, and B3.Here, it is assumed that the feature points are extracted from the imageB2 using brightness differences and the like in an edge of the notebookand/or pencil so as to start the detection of the angle θo. Then, in thecase where the user 10 moves the cellular phone 20 to the left handside, a part of the notebook disappears from the image and the wholeimage of the pencil is displayed, as shown in the image B1. Conversely,in the case where the user 10 moves the cellular phone 20 to the righthand side, as shown in the image B3, the mug appears in the image. Then,tracking of plural feature points are executed in parallel, inaccordance with the movement of the opposite-side camera 22. With this,even when a part of the feature points disappears from the image, it ispossible to detect the angle θo from other feature points. In addition,when a new feature point is detected by the object which appears in theimage, it is added as a target for tracking feature points. In thismanner, by tracking the feature points in the image captured by theopposite-side camera 22, in other words, by comparing the image featuresextracted from the second image IM2 to the image features extracted fromthe past images, it is possible to detect the angle θo with highresolution. Note that as an object captured by the opposite-side camera22 is arbitrary, the stability of the feature detection is inferior tothe detection by the user-side camera 21 which constantly captures faceimages.

In other words, as the user-side camera 21 constantly captures the faceof the user 10, the detection of the marker is stable. However, as theamount of movement is small, the method is not suitable for detectingthe angle θo in details. On the other hand, as the image captured by theopposite-side camera 22 largely varies, the amount of movement of thefeature points is large and the angle θo can be calculated in details.However, as the captured scenes are arbitrary, there are possibilitiesthat the precision with which to detect the marker becomes unstable.

In this manner, as the user-side camera 21 and opposite-side camera 22have both merits and demerits in detection of the angle θo respectively,it is possible to improve the detection with precision by complementingthe operations each other. For example, when the detection of thefeature points from the image ends in failure and until theopposite-side camera 22 recovers the detection of the feature points,the image captured by the user-side camera 21 may be used for detectingthe angle θo.

In addition, for example, the mean values of the angle θo calculatedrespectively from the images captured by the user-side camera 21 and theopposite-side camera 22 may be used. In addition, for example, among theangles θo calculated respectively from the images captured by theuser-side camera 21 and the opposite-side camera 22, unique values maybe excluded for the use, judging from the past history, a predeterminedthreshold value or the like.

The rotation angle φo to the vertical direction can also be detectedlikewise the rotation angle θo to the horizontal direction. Since theimage of the user-side camera 21 largely varies compared to motion tothe horizontal direction, it is considered that the detection precisionis slightly improved. Note that the merit-demerit relationship betweenthe user-side camera 21 and the opposite-side camera 22 is the same asthe one in which the direction is changed to the horizontal direction.Therefore, in order to improve the position detection precision,cooperative processing through complementary operations using imagescaptured by two cameras becomes effective.

In other words, the direction calculation unit 35 calculates the angleθo, φo, that is, the direction of the cellular phone 20 using thecomparison results by the position comparison unit 32 b and the positioncomparison unit 37 b.

The cooperative processing through captured images by two cameras isalso effective at detecting the distance ro. In other words, it ispossible to obtain the distance ro from the second image IM2 using thesize variations of the marker graphics. It is probable that thedetection becomes unstable because the marker graphics are extractedfrom arbitrary scenes. However, as there are possibilities that themarker extraction from the first image IM1 may end in failure due tosome reasons, the distance detection from the second image IM2 iseffective as the backup.

In addition, the orientation of the cellular phone 20 can be detected bythe position comparison or the shape comparison of the marker graphics.FIG. 7 shows an image captured by the user-side camera 21 when theorientation of the cellular phone 20 is changed. In FIG. 7, an image D1is an image when the cellular phone 20 is rotated by the angle θc to thehorizontal direction. In the image D1, the user 10 is displaced to thelateral direction (right side in this example). The image D2 is an imagewhen the cellular phone 20 is rotated by the angle φc to the verticaldirection. In the image D2, the user 10 is displaced to the longitudinaldirection (up in this example).

Thus, it is possible to detect an orientation of the cellular phone 20by obtaining the amount of displacement of the marker graphics position(for example, the edge of the shoulder). In addition, by focusingattention on the shape of the marker graphics, the graphics become flatwith the variations of the orientation, and for example, a circle ischanged to an oval. Thus, it is possible to detect the orientation ofthe cellular phone 20 from variations of the shape of the markergraphics. Likewise, it is possible to detect the orientation of thecellular phone 20 from an image captured by the opposite-side camera 22.

In other words, the orientation calculation unit 39 calculates theangles θc, φc, that is, the orientation of the cellular phone 20 usingthe comparison results by the position comparison unit 32 b and shapecomparison unit 32 c and the comparison results by the positioncomparison unit 37 b and shape comparison unit 37 c.

Note that there is a case where using, as a marker, lighting in abackground of the face of the user 10 is effective when performing thedetection from the first image IM1 captured by the user-side camera 21.In such a case, the marker graphics may be extracted by giving the firstimage IM1 to the scene feature point extraction unit 36.

The display control unit 41 displays, on a display unit 44, an imagestored in an image memory unit 43 selected by the user 10. In addition,the display control unit 41 controls a display of the image (displayedimage) displayed on the display unit 44 based on the distance rocalculated by the distance calculation unit 34, the angles θo, φocalculated by the direction calculation unit 35, and the angles θc, φccalculated by the orientation calculation unit 39. Note that in theimage memory unit 43, plural images captured from different directionsand having different sizes are stored in advance.

FIG. 8 is a diagram describing the control of enlargement or reductiondisplay. Here, a display image E is initially displayed on the displayunit 44. The display control unit 41 performs control of displaying thedisplay image E shown in FIG. 8 by enlarging or reducing the image inaccordance with the distance ro calculated by the distance calculationunit 34. In other words, for example, when the cellular phone 20 ismoved by the user 10 and as a result, the distance ro becomes shorter,the display control unit 41 controls the display by enlarging the imageso as to display the enlarged image as a display image E1 shown in FIG.8. On the other hand, for example, when the distance ro becomes longer,the display control unit 41 controls the display by reducing the imageso as to display the reduced image as a display image E2 shown in FIG.8. In addition, the display control unit 41 controls, with the anglesθc, φc calculated by the orientation calculation unit 39, a centerposition in the case where the display image E is enlarged or reduced.

FIG. 9 is a diagram describing the control of a modified display to adisplayed direction. Here, the display image E is initially displayed onthe display unit 44. The display control unit 41 performs control of adisplay by changing the displayed direction of the display image E shownin FIG. 9 in accordance with the angles θo, φo calculated by thedirection calculation unit 35. In other words, for example, when thecellular phone 20 is moved by the user 10 and is moved to the rightdirection (the angle θo is in a positive direction), the display controlunit 41 controls the display of the image viewed from the rightdirection as a display image E3 shown in FIG. 9. On the other hand, whenthe cellular phone 20 is moved to the left direction (the angle θo is ina negative direction), the display control unit 41 controls the displayof the image viewed from the left direction as a display image E4 shownin FIG. 9. In addition, for example, when the cellular phone 20 is movedupward (the angle φo is a positive direction), the display control unit41 controls the display of the image viewed from below as a displayimage E5 shown in FIG. 9. On the other hand, when the cellular phone 20is moved below (the angle φo is a negative direction), the displaycontrol unit 41 controls the display of the image viewed from above as adisplay image E6 shown in FIG. 9.

As described above, the cellular phone 20 is moved from the initialposition by the user 10, and the display control unit 41 controls thedisplay of the displayed image in accordance with the amount of movementfrom the initial position (relative value). However, the presentinvention is not limited to such control. For example, the displaycontrol unit 41 may control the displayed image in accordance with thedistance ro calculated by the distance calculation unit 34, the valuesof the angles θo, φo (absolute values) calculated by the directioncalculation unit 35, and the values of the angles θc, φc (absolutevalue) calculated by the orientation calculation unit 39.

In addition, the display control unit 41 displays, on the display unit44, an image stored in the image memory unit 43. The image is notlimited to the image stored in the image memory unit 43 provided in thecellular phone 20. For example, the display control unit 41 may controldisplay of an image obtained via networks, such as the Internet, forexample an image in on-line shopping.

In addition, when the display control unit 41 controls display of adisplay image, it may control the display of a displayed image bystoring a single image in the memory 42 and perform image processing onthis image without using plural images respectively captured fromdifferent directions and having different sizes.

FIG. 10 is a block diagram showing configuration other than theposition/orientation detection circuit. In order to describe a positionof the cellular phone 20 in the rotating coordinate system in which theuser 10 is a rotation center, as described in FIG. 6, using the secondimage IM2 captured by the opposite-side camera 22 is preferable fordetecting the direction of the cellular phone 20. From these viewpoints,the circuit scale of the configuration in FIG. 10 is largely reducedcompared to the configuration shown in FIG. 4.

In other words, in the position/orientation detection circuit 30A ofFIG. 10, the position comparison unit 37 b and direction calculationunit 35A detect the angles θo, φo from the second image IM2 captured bythe opposite-side camera 22. In addition, as the position comparisonunit 37 b calculates differences between the positions of the markergraphics, the orientation calculation unit 39A detects the orientationsθc, φc based on the resulting calculation. On the other hand, as themarker extraction from face images is stable, the distance ro isdetected using the first image IM1 captured by the user-side camera 21.

FIG. 11 is also a block diagram showing configuration other than theposition/orientation detection circuit. The marker as a key fordetection can detect a particular object with precision rather than anarbitrary subject. Therefore, the circuit scale in the configuration ofFIG. 11 is largely reduced compared to the configuration shown in FIG. 4by prioritizing the processing which extracts a marker from a faceimage.

In other words, in the position/orientation detection circuit 30B ofFIG. 11, the position comparison unit 37 b and direction calculationunit 35A execute the detection of the angles θo, φo from the secondimage IM2 captured by the opposite-side camera 22. On the other hand,the distance ro and orientations θc, φc are detected by the first imageIM1 captured by the user-side camera 21, and a face image is detected asthe marker graphics by the size comparison unit 32 a, distancecalculation unit 34A and orientation calculation unit 39B.

Although the case where the user 10 moves the cellular phone 20 by thehand is described in the present embodiment, the method of movingapparatuses is arbitrary, and the method in the present invention is notlimited to such.

In addition, precisely speaking, positions of the two cameras aredifferent. However, as such cameras are normally small in size, thedifference can be ignored. Even when it can not be ignored, the positionmay be corrected by making conversions of the geometric positions of thetwo cameras, and there may be a case when the mean values of the twocamera positions can be applied.

In the present embodiment, the optical axis of the user-side camera 21matches that of the opposite-side camera 22. When the optical axis ismatched, the two cameras 21 and 22 geometrically have the easiestpositional relationship. In other words, this is an idea for making thegeometric conversions between the two cameras 21 and 22 simpler. Notethat the optical axes of the two cameras do not have to be matched, andas long as the geometric relationship is known, the two cameras may havean arbitrary positional relationship.

In addition, although the present embodiment uses two cameras, forexample, a single wide angle camera may capture an image covering awider area. In other words, among two captured images, one of them hasonly to include a known object as image features.

Although only an exemplary embodiment of this invention has beendescribed in detail above, those skilled in the art will readilyappreciate that many modifications are possible in the exemplaryembodiment without materially departing from the novel teachings andadvantages of this invention. Accordingly, all such modifications areintended to be included within the scope of this invention.

INDUSTRIAL APPLICABILITY

As the present invention can be realized by a simple structure havingtwo cameras, it is possible to introduce it to portable apparatuses withease. By sequentially displaying, on a display, images in accordancewith a position and an orientation of the apparatus, even in the casewhere the display area is small, the user can watch, through thedisplay, the whole subject and observe the subject with more reality asif he/she touched it. For example, with this, it is possible tosubstantially improve practical utility of on-line shopping. Inaddition, the present invention can be applied to medical use for whichan image captured on a spot and computer graphics can be combined and touse in museums, such as digital archives.

1. A display control method for controlling a display of a displayedimage based on a position of an apparatus provided with a first cameraand a second camera which are in a known geometric relationship, saiddisplay control method comprising: capturing, by the first camera, afirst image including an object with a known image feature; capturing,by the second camera, a second image including a scene in the vicinityof the object; extracting the image feature of the object from the firstimage; calculating a distance from the object to the apparatus and adirection of the apparatus viewed from the object, by comparing theextracted image feature of the object to the known image feature of theobject; extracting an image feature of the second image from the secondimage; calculating a distance from the object to the apparatus and adirection of the apparatus viewed from the object, by comparing theextracted image feature of the second image to an image featureextracted from a past image which has been captured by the secondcamera; and controlling the display of the displayed image based on thedistance and direction calculated based on the first image and thedistance and direction calculated based on the second image.
 2. Thedisplay control method according to claim 1, wherein the object is aface of a user who uses the apparatus.
 3. The display control methodaccording to claim 1, wherein said controlling of the display of thedisplayed image includes calculating the distance from the object to theapparatus and the direction viewed from the object using, in acomplementary manner, the distance and direction calculated based on thefirst image and the distance and direction calculated based on thesecond image, and controlling the display of the displayed image basedon the calculated distance and direction.
 4. The display control methodaccording to claim 3, wherein said controlling of the display of thedisplayed image includes controlling the display of the displayed imagebased on the distance and direction calculated based on the first imagein the case where the image feature of the second image can not beextracted and until the image feature of the second image can beextracted.
 5. The display control method according to claim 1, furthercomprising calculating an orientation of the apparatus with respect tothe object using at least one of the image feature of the objectextracted from the first image and the image feature of the second imageextracted from the second image, wherein controlling the display of thedisplayed image includes controlling the display of the displayed imagebased on the calculated distance from the object to the apparatus, thecalculated direction of the apparatus, and the calculated orientation ofthe apparatus.
 6. A display control device for controlling a display ofa displayed image based on a position of an apparatus provided with afirst camera and a second camera which are in a known geometricrelationship, said display control device comprising: a first featureextraction unit operable to extract an image feature of an object from afirst image including a known image feature of the object, the firstimage being captured by said first camera; a first calculation unitoperable to calculate a distance from the object to said apparatus and adirection of said apparatus viewed from the object, by comparing, to theknown image feature of the object, the image feature of the objectextracted by said first feature extraction unit; a second featureextraction unit operable to extract an image feature of a second imagefrom the second image including a scene in the vicinity of the object,the second image being captured by said second camera; a secondcalculation unit operable to calculate a distance from the object tosaid apparatus and a direction of said apparatus which is viewed fromthe object, by comparing the image feature of the second image extractedby said second feature extraction unit to an image feature extractedfrom a past image which has been captured by said second camera; and adisplay control unit operable to control the display of the displayedimage based on the distance and direction calculated by said firstcalculation unit, and the distance and direction calculated by saidsecond calculation unit.
 7. A portable apparatus provided with a firstcamera and a second camera which are in a known geometric relationship,said portable apparatus comprising a display control device operable tocontrol a display of a displayed image based on a position of saidportable apparatus including: a first feature extraction unit operableto extract an image feature of an object from a first image including aknown image feature of the object, the first image being captured bysaid first camera; a first calculation unit operable to calculate adistance from the object to said apparatus and a direction of saidapparatus viewed from the object, by comparing, to the known imagefeature of the object, the image feature of the object extracted by saidfirst feature extraction unit; a second feature extraction unit operableto extract an image feature of a second image from the second imageincluding a scene in the vicinity of the object, the second image beingcaptured by said second camera; a second calculation unit operable tocalculate a distance from the object to said apparatus and a directionof said apparatus which is viewed from the object, by comparing theimage feature of the second image extracted by said second featureextraction unit to an image feature extracted from a past image whichhas been captured by said second camera; and a display control unitoperable to control the display of the displayed image based on thedistance and direction calculated by said first calculation unit, andthe distance and direction calculated by said second calculation unit,wherein said display control device is operable to control the displayof the displayed image based on a position of said portable apparatus.8. A program for controlling a display of a displayed image based on aposition of an apparatus provided with a first camera and a secondcamera which are in a known geometric relationship, said program causinga computer to execute: extracting an image feature of an object from afirst image including a known image feature of the object, the firstimage being captured by the first camera; calculating a distance fromthe object to the apparatus and a direction of the apparatus which isviewed from the object, by comparing the extracted image feature of theobject to the known image feature of the object; extracting an imagefeature of a second image from the second image including a scene in thevicinity of the object, the second image being captured by the secondcamera; calculating a distance from the object to the apparatus and adirection of the apparatus viewed from the object, by comparing theextracted image feature of the second image to an image featureextracted from a past image which has been captured by the secondcamera; and controlling the display of the displayed image based on thedistance and direction calculated by the first image, and the distanceand direction calculated by the second image.